While many studies about COVID-19 policies published (Phillips, Zhang, and Petherick 2021; Hale et al. 2021; Flaxman et al. 2020; Haug et al. 2020), we still don’t know all that much about how effective policies are in containing the virus.
A lot of the problem, we argue has to do with measurement, a necessary prior step to causal inference.
In this paper, we present new COVID-19 policy intensity scores that are robust to measurement error and permit accurate interpretation of the effect of COVID-19 policies on other outcomes.
For example, lockdowns are often seen as a singular COVID-19 restriction. But they are actually a combination of different policies:
Restrictions on movement of various kinds.
Can you leave your house?
Can you travel to stores?
What about emergency travel?
Restrictions on gathering
Can you meet with family members?
Coworkers?
Recreational activities?
We can use simple binary classifications for policies, i.e., 1 if any policy imposed.
Policies are often highly correlated because they a part of an over-arching policy framework.
The Flaxman et al. (2020) study had severe problems with this—reclassifying lockdowns or moving the dates could switch the weight of different policy indicators dramatically .
The inference problem is similar to interpreting control variables in a regression model: we only see the direct effects, which can be very misleading (Hünermund and Louw, n.d.; Keele, Stevenson, and Elwert 2020).
These issues can be seen in existing work as policies have implausible reported associations such as lockdowns increasing COVID-19 infections.
Paper used a very informative prior on the effect of COVID-19 policies on reducing infections:
“our prior places 80% of its mass on positive effects, reflecting a belief that NPIs are more likely to reduce transmission than to increase it.”
Results without this prior show positive associations between lockdowns and COVID-19.
We can fix this measurement issue by isolating the variation in disparate policy records that is directed at a given policy domain (business restrictions, social gatherings, masks, etc).
Assume each policymakers has an ideal point \(x_i\) for a certain policy domain that will maximize their utility by minimizing distance to policies \(Y_{ij}\) where each policy has a Yes \(Y_j\) and No \(N_j\) point in the ideal point space:
\[ U_i = \lVert x_{it} - N_j \rVert^2 - \lVert x_{it} - Y_j \rVert^2 \qquad(1)\]
For each iteration, draw a value from the following distributions for each latent parameter–time-varying ideal points \(x_{it}\), policy yes points \(Y_j\), no points \(N_j\), and policymaker utility \(U_{itj}\):
\[\begin{align} x_{i(t=1)} &\sim \text{Normal}(0,1)\\ x_{it} &\sim \text{Normal}(x_{i(t-1)},0.25)\\ N_j &\sim \text{Exponential}(1)\\ Y_j &\sim N_j + \text{Exponential}(2)\\ U_{ijt} &= \lVert x_{it} - N_j \rVert^2 - \lVert x_{it} - Y_j \rVert^2 \end{align}\]For each value of a latent parameter, draw the following observed data: discrete and continuous policies \(Y_{ij}\) and COVID cases \(I_{it}\) as a function of utilities \(U_{ijt}\) and ideal points \(x_{it}\):
\[\begin{align} Y_{i(j<6)} &\sim \text{Normal}(U_{ijt}, 0.5) \\ M_{i(j>5)} &\sim \text{Normal}(0, 0.5) \\ Y_{i(j>5)} &\sim \text{Bernoulli}(q(U_{ijt} + M_{ij}))\\ I_{it} & \sim \text{Binomial}(q(-10 - 2 x_{it}), 10000000) \end{align}\]\[ Pr(Y_{ij}) = g(\delta_j^{'} \hat{x_i} - \beta_j) \qquad(2)\]
Then we run regressions on COVID cases \(I_{it}\):
\[ I_{it} = \alpha_1 + \beta_1 Y_{ij} \qquad(3)\]
\[ I_{it} = \alpha_2 + \beta_2 \hat{x_{it}} \qquad(4)\]
Which coefficient will match the simulation effect of ideal points on policy outcomes?
To estimate these indices, we combine CoronaNet and Oxford datasets on COVID-19 restrictions.
We map dozens of indicators onto six policy domains: masking, health monitoring, health resources, school restrictions, business restrictions, and social distancing.
We estimate these using time-varying Bayesian item response models (idealstan) using the same specification as Equation 2.
Estimates of policy intensity/ideal points \(\hat{x_{it}}\) include uncertainty.
A very boring regression table:
Best way to get the latest version of our scores is with our R package CoronaNetR on CRAN:
Paper draft available here: